Delivery and licensing models such as Software as a Service (SaaS), Platform as a Service (PaaS), Infrastructures as a Service (IaaS), other similar on-demand software such as subscription based models, etc. have increasingly gained popularity, especially in business software applications. With these models, users may access the provided services using a client (e.g., a thin client) via web browsers or mobile applications on mobile computing devices. More such models have been implemented under the end-to-end computer networking architecture where features or services reside in the end nodes of the computer network, while the users reside on the other end of the computer network. An end-to-end computer networking architecture addresses the shortcomings of conventional software delivery and licensing models because the provided services and applications may need to be modified, and new features, applications, or services may be added to the provided services. Each modification or addition of the provided services incurs costs. Yet for the aforementioned services, not all users use or are licensed to use all the services. For example, a subscriber under a subscription based license may be authorized only a small subset of all the services provided.
Testing the functional and non-functional behaviors of these systems has been a challenging task. A team of engineers often needs to develop test cases to validate the behaviors of a system, and these test cases need to be orchestrated and are often performed in series in the computing platform because one test case may require one or more specific environment and/or system states that may be incompatible or inconsistent with another test. For example, testing a billing system or module for a subscription based service may require a team of engineers to develop a large number of test cases that will be executed to generate various test results.
Conventional approaches initiate test sessions in which these test cases are executed in series. More specifically, when a first test case is executing to generate various test results, the other test cases need to wait for the completion of the first test case, validation or verification of the software modules involved in the first test case, and subsequent removal of the test results and reset of various system and environment states before another test case may be started. One immediate drawback of these serial approaches is that testing a software system inevitably requires a long time. Practical experiences have shown that testing a software system typically takes hundreds of engineer-weeks, if not longer, to complete, where an engineer-week indicates a week regarded in terms of the amount of work that may be done by one engineer within this period.
Another drawback is that the test platform including one or more computing nodes and their environments need to be restored and set to the requisite states prior to the initiation of another test session for another test case. The test results generated for the previous test case will be removed during the restore and reset. If there is another issue that is discovered after the restore and reset, the test results have already been removed so any debugging and troubleshooting have to be performed without the supporting test results. This lack of test results increases the difficulties and challenges in effectively debugging and troubleshooting a software system under test.
Some other conventional approaches attempt to address these difficulties by orchestrating the test cases into buckets or groups by an engineer or an architect where test cases sharing the same states or characteristics may be grouped into a single bucket and executed together. Although these conventional approaches improve the overall efficiencies, these buckets are nevertheless executed in series. As a result, these conventional approaches still suffer from the same deficiencies as described above.
Therefore, there exists a need for methods, systems, and computer program products for implementing end-to-end automation of software services to address at least the aforementioned challenges.